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METHOD AND APPARATUS FOR SCALABLE PROCESS FLOW ( 
LOAD BALANCING OF A MULTIPLICIl Y OF PARALLEL PACKET 
PROCESSORS IN A DIGITAL COMMUNICATION NETWORK 



Field of the Invention 

The present invention relates to an apparatus that monitors data packets 
transmitted on a data network and processes the data packets. More 
particularly, the present invention relates to an apparatus that classifies the 
5 data packets and associates the classified data packets with a particular data 
flow. Then, the apparatus selects a particular packet processor from a plurality 
of packet processors to process the classified data packets that correspond to 
the particular data flow. Also, the apparatus determines which one of the 
plurality of packet processors to select from as the particular packet processor 

10 to process a particular process flow, based on information relating to the 
processing loads of the processors. By designating certain packet processors 
to process data packets corresponding to data flows based on the load of the 
processors, the apparatus can ensure that the processing loads of the plurality 
of packet processors is balanced and allows for wire speed processing of the 

15 process flows. Furthermore, the present invention relates to a method and a 
software routine employed by the apparatus. 

Background of the Invention 

In a digital communication network (e.g. the internet, wide area 
network ("WAN")? local area network ("LAN"), service aware network 
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applications such as internet video conferencing and internet voice 
conferencing is improved. Also, the network components can quickly process 
the data packets to determine if they are authorized to be transmitted to the 
destination computer, and if they are not, the network components discard the 
5 datapackets. As a result, the security of the network is greatly enhanced. The 
ability to handle the above mentioned situations on a higher level than the 
basic packet level alone, will further enhance the desired system level 
performance. 

Before proeessing a data packet, a network eomponent must "ctesify" 
,0 the data packet according to various characteristics of the data packet and/or 
the data contained in the packet Then, the network component processes the 
data packet based on its classification. Furthermore, the classification of .he 
data packet enables the data packet to be associated widt the other data packets 
belonging to a particular st^an, of packets. As a result, data packets 
,5 belonging to a certain stream or "process flow" can all be processed by the 
same packet processing «nt within the network component. By processing 
,he data packets belonging to tire same process flow, the packet processing 
unit can process tire data packets in a more efficient mamier. 

A data packet is usually classified by evaluating tire infotmation 
20 contained in the data packet header. For example, if tire data packet contains 
the header HDR shown in Fig. 1, a network component may classify d-e data 
packet as a first type of data packet if tire source IP address falls ^rttitin a first 
range of sour^ IP addresses, tire destination IP address falls wittrin a first 
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range of destination IP addresses, the protocol data falls within a first range of 
protocol data values, the source port data falls within a first range of source 
port data values, and the destination port data falls within a first range of 
destination port data values. On the other hand, the internet component may 
5 classify the data packet as a second type of data packet if the source IP 
address, destination IP address, protocol data, source port data, and destination 
port data respectively fall within a second range of source IP addresses, a 
second range of destination IP addresses, a second range of protocol data 
values, a second range of source port data values, and a second range of 

1 0 destination port data values. 

Each group of data value ranges by which a data packet is classified 
may be considered to be a "rule". Thus, in the examples above, the data 
packet is classified as the first type of data packet if its header HDR satisfies a 
first rule defined by the first range of source IP addresses, destination IP 
15 addresses, protocol data values, source port data values, and destination port 
data values. On the other hand, the data packet is classified as the second type 
of data packet if its header HDR satisfies a second rule defined by the second 
range of source IP addresses, destination IP addresses, protocol data values, 
source port data values, and destination port data values. Furthermore, the 
20 data packets may be classified based on a subset of the data value ranges 
mention above, additional data value ranges, or different types of criteria. 

After the data packet is classified, the network component is able to 
determine how to handle or process the data. For instance, based on the 
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path so that they ,uicMy reach their intended destittation computer, may be 
evaluated to determine if the data packets are authorized to he received and 
fi^er processed by the network component, may he prevented .om hemg 
forwarded on the network, may he processed in a particular manner, etc. 
,0 Accordingly, the network component classifies incoming data packets 

aata packet headers HDR and processes the data packets based on th=,r 

classification. 

Since the network component must classify each and every data packet 
, 5 that it receives, it should ideally classify the data packets at a speed «rat e.uals 

-.^,e speed". By classifying the data packets as quickly as they are recetved, 
aata packets do no. become "hottlenecked" at the input of the network 
eo^ponen, and the overall operational speed of the network is not degraded. 

However, as the speeds a. which networks are capable of transmi«ing 
aata packets increase, the speeds at which network components must be able 
.„ Classify and process data packets must likewise increase. For example, on a 
high speed Sonet network that is capable of transmitting ten gigabits per 
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..ond, da.a pacR^s can b. »ans,nitt=d a, a ,a« of 30 m«Uo„ packets pe. 
second, and on a M. duplex Une. data packets can be .ransn^itted a. about 60 
„i,Hon packets per second. Thus, network components must be able .o 
classify data packets at extraordinary speeds. 
5 In addition to classifying data packets a. high speeds, network 

eontponents must be able to classiiy the data packets based on several 
p3ran,eter fields within the packet Currently, classifying the data packets 
based on the several parameter fields results in Cassifymg the packets based 
on hundreds of rules, TT^us, to properly classity the incoming data packets 
,0 Without creating a bottleneck at Ute input of the network component, the 
component must determine which rule of the hundreds of rules corresponds to 
each of the incoming data packets and must make such determination at a very 
high speed. Furthermore, as the number of network users and the number of 
different services available on the network increase, the number of nrles drat 
,5 will need to be evaluated by standard network components is expected to grow 

components will need to classify data packets according to an extremefy large 
number of rules at incredible speeds. 

in Ught of the above demands, network components must be designed 
20 that can efficiently classify and process the data packets that at. transmitted at 
very high speeds. In the example described above in which classified data 
packets are classified and stored in particular queues based on thetr 
Classification, the processing speed of Ute network component is somewhat 
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enhanced. However, the network component is only able to perform basic 
operations on the data packets travelling on the network and cannot associate 
groups of data packets together so that they can be processed more efficiently. 
Furthermore, the network component is unable to associate data packets 
5 travelling downstream in the network with corresponding data packets 
travelling upstream in the network. As a result, the downstream data packets 
and upstream data packets are processed separately in accordance with 
separate processes, and the overall efficiency of the network components in 

the network is decreased, 
10 Also, in the above example, the loads of the processors that process the 

data packets within the network component are not monitored or evaluated. 
As a result, the various process flows of the data packets camiot be allocated 
among the processors such that the processing loads of the processors are 
evenly balanced. By not allocating the process flows of the data packets to the 
15 various processors such that the loads of the processors are balanced, the 
overall efficiency of the network component is degraded, and the data and/or 
applications received by the network component camiot be processed in real 
time or at wire speed. Thus, a substantial need exists for a system in which 
process flows of data packets are allocated to packet processors in such a 
20 manner that the loads of the processors are evenly balanced. 

Network systems have been developed that generally balance the 
processing loads of various processors in certain applications. However, such 
systems do not suggest balancing the processing load of processors within a 
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network component by designating particular processors to handle certain 
process flows of data packets. 

For example, U.S. Patent No. 6,026,425 to Suguri et al. ("the '425 
patent") discloses a system that estimates the processing loads of several 
5 nodes and determines whether or not a particular node is capable of accepting 
a task based on its processing load. Such patent is incorporated herein by 
reference for all purposes and a brief description of the disclosed system is 
provided below. 

As shown in Fig. 5 of the '425 patent, the system contains a plurality 
10 of nodes NODEl to NODEn for processing various tasks. Also, each of the 
nodes NODEl to NODEn contains a load balancing apparatus 10 and is 
connected to a logical ring network 60B. New tasks to be processed by one of 
the nodes NODEl to NODEn are input to the logical ring network 60B and 
initially supplied to the first node NODEl. Upon receiving a new task, the 
15 load balancing apparatus 10 within the node NODEl determines whether or 
not the processing load of the node NODEl is greater than a threshold value. 
If the processing load is less than the threshold value, the new task is 

processed by the node NODEl . 

On the other hand, if the processing load is greater than the threshold 
20 value, the task is not processed by the first node NODEl and is output to the 
logical ring network 60B. In such case, the new task is then suppHed to the 
second node NODEl, and the load balancing apparatus 10 within the node 
NODE2 determines whether or not the processing load of the node NODE2 is 
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U.S. Patent No. 6,006,259 to Adelman et al. ("the '259 patent") 
generally provides for balancing of loads among end-nodes within a network. 
For that prurpose, a computer system is required in each of the end-nodes. 
These systems, otherwise known as a "web switch", allow an end-user to have 
5 the perception that one end-node is working for him or her, while in reality a 
plurality of such end-nodes are performing certain applications. This solution 
does not address the issue of balancing the loads of processors based on the 
process flows of the data packets transmitted over the network in a network 
element, working independently of other network elements. The '259 patent 
10 therefore cannot ensure operation at wire speed, especially when scaling is 
required, as the distribution of the end-nodes may be significant. Thus, the 
'259 patent cannot be employed in a system to monitor the operation of a node 
of a network and efficiently manage the operation of a node to ensure that it 
operates in an extremely efficient manner. Moreover, the '259 patent does 
15 address how the load is divided between the end-nodes. The disclosure of the 
U.S. Patent No. 5,983,281 to Ogle et al. ("the '281 patent") deals with 
balancing the loads between gateways handling the transmission of packets 
between end-nodes. As described in the patent, a first end-node sends a packet 
to a second end-node through a plurality of gateways, and a determination of 
20 which gateways should be used is done such the load on the gateway system is 
reduced. While somewhat different in nature from the '259 patent described 
above, both the '259 and '281 patents relate to the global distribution of loads 
between independent systems. Therefore, when wire speed operation is 
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required (e.g., when dautomatic decision making is required), the distributed 
solutions disclosed in the '259 and '281 patents do not provide an acceptable 
answer to the issue of load balancing. The '259 patent and the '281 patent are 
incorporated herein by reference for all purposes. 

Also, U.S. Patent No. 5,841,775 to Huang ("the '775 patent") is 
incorporated herein by reference for all purposes and discloses a network 
system in which data packets can be routed from a certain input terminal to a 
certain output terminal via one of several possible routing paths. Furthermore, 
the data packets are sequentially transferred from the certain input terminal to 
3 the certain output terminal by alternatively routing the data packets over the 
several possible routing paths. As a result, the loads of the processors 
contained within the various routing paths are relatively balanced. Also, if a 
particular path of the several possible routing paths contains a malfunction or 
is overloaded, data packets are not supplied to the output terminal via the 
.5 particular path and are routed to the output terminal via one of the remaining 
routing paths. 

However, the '775 patent does not teach a system that balances the 
load among various processors by designating certain processors to process 
data packets belonging to a particular flow. T^us, the disclosed system is not 
20 well suited for processing data packets in real time or at wire speed. 

U.S. Patent No. 5,825,860 to Moharran ("the '860 patent") and U.S. 
Patent No. 5,271,003 to Lee et al. ("the '003 patent") generally relate to 
systems in which data is selectively supplied to various processors and are 
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overcoming the problems described above. 
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selecting a first selected processor of said packet processors to process said 
first data packet, wherein said first selected processor is selected based on 
processing load values respectively corresponding to processing loads of said 
packet processors. 

5 In order to additionally achieve the above and other objects, a software 

program contained on a computer readable medium is provided. The software 
program enables a controller to perform a routine, comprising: (a) receiving a 
first data packet; (b) determining whether or not any packet processors have 
, fi been previously selected to process said first data packet based on a 

rj 10 classification of said first data packet; (c) when none of said packet processors 

I'll 

ry has been previously designated to process said first data packet, selecting a 

'''^ first selected processor of said packet processors to process said first data 

packet, wherein said first selected processor is selected based on processing 

ri I 

1.5 load values respectively corresponding to processing loads of said packet 

Q 15 processors. 

In order to even further achieve the above and other objects, an 
apparatus for distributing processing loads is provided. The apparatus 
comprises: a controller; and a plurality of packet processors coupled to said 
controller, wherein said controller receives a first data packet and determines 
20 whether or not any of said packet processors have been previously selected to 
process said. first data packet based on a classification of said first data packet, 
wherein, when none of said packet processors has been previously designated 
to process said first data packet, said controller selects a first selected 
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processors of said packet processors to process said unclassified data packets 
based on load information, wherein said packet processors respectively have 
processing loads and said load information corresponds to said processing 
loads. 

5 In order to even fiirther achieve the above and other objects, an 

apparatus is provided. The apparatus comprises: a plurality of packet 
processors; and a controller, wherein said controller receives input data 
packets from a stream of data packets and determines which of said input data 
packets are unclassified data packets, wherein said unclassified data packets 

10 have unclassified data packet classifications and none of said packet 
processors has been previously designated to process data packets having any 
of said unclassified data packet classifications, wherein said controller selects 
designated processors to process said unclassified data packets based on load 
information, and wherein said packet processors respectively have processing 

1 5 loads and said load information corresponds to said processing loads. 

Brief Description of the Drawings 

The above and other objects and advantages of the present invention 
will become more apparent by describing in detail preferred embodiments 
thereof with reference to the attached drawings in which: 
20 Fig. 1 illustrates an example of the format of a header of a data packet; 

Fig. 2 illustrates an example of the structure of a network monitoring 
and classifying system according to an embodiment of the present invention; 
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Fig. 3 illustrates an example of classifying processing loads of the 

packet processors shown in Fig. 2; 

Fig. 4 inustta.es ^ example of assigmng new process flows to fte 

packet processors shown in Fig. 2; 
5 Fig. 5 illusttates a second example of classifying processing loads of 

the packet processors shown in Fig. 2; 

Fig. 6 illustrates a third example of classifying processing leads of Ae 

packet processors shown in Fig. 2; and 

Fig. 7 illustrates a fourth example of classifying processing loads of the 

10 packet processors shown in Fig. 2. 

The following descripUon of the preferred embodiments discloses 
.peeiftc configurations, components, and process steps. However, dte 
p^ferred embodiments are merely examples of the present invention, and 

describe such embodiments »d to provide an overall understanding of the 

■ ,i„„ Accordingly, one skilled in the art will readily recognize 

present invention. Accorauigiy, 

that the present invention is not limited to the specific embodiments described 
wow Furthermore, the descriptions of various configurations, components, 

in the art are omitted for the sake of clarity and brevity. 
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The subject matter disclosed in the present application is generally 
related to the subject matter disclosed in an application that is entitled 
•■Apparants and Method for Wire-Speed Classification and Pre-Processing of 
Data Packets in a Full Duplex Network". The application was filed on April 
5 3, 2000, invented by Michael Ben-Nun. Sagi Ravid. and Offer Weil, and 
assi^ed to the assignee of the present application. Such application is 
incorporated herein by reference for all purposes. 

The present invention relates to a monitoring and classifying system 
that can simultaneously monitor the upstream and downstream flow of data 
,0 packets. An illustrative, non-limiting embodiment of the system 200 is shown 
i„ Fig. 2 and comprises a cential conttol unit ("CCU 205") and a plurality of 
packet processors PP. to PPN. Also, the CCU 205 comprises physical access 
^its 210 and 220. data path units 230 and 240. a header processor 250. and a 
classifymg unit 260. In a preferred embodiment, the system 200 a, least 
, 5 partially constitutes a network component, and the network component can be 

alternative embodiment, the network component can be directiy coupled to 
more than two nodes of the network. 

The physical access unit 210 comprises standard layer one and layer 
,0 two components ti>at input and capture data packets travelling upstieam ftom 

.reared ftom one or more commercially available hardware units known as 
physical access ("PHY") devices, and media access control ("MAC") devices. 
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230 or 240 to inform the classifier 260 to. the corresponding d«a packet does 
not correspond to any of tlte predetermined rules and that there is no need to 
maintain a process fiow for such packets. 

The classifier 260 inputs the headers HDR of the data packets and 
5 evaluates the headers HDR to determine a specific "flow" corresponding to 
each of dte data packets. A specific flow relates to the communications or the 
transfer of data packets between two computers (or nodes) of the network. For 
instance, data packets transmitted from a first source computer (or node) to a 
first destination computer (or node) of the network belong to a first flow, and 
,0 data packets transmitted from the first sou.ce computer (or node) to a second 
destination computer (or node) belong to a second flow. By evaluating the 
. contents of the HDR of a data packet, the classifier 260 can determine the 
specific flow conesponding to the data packet. 

,n addition, in a flrll duplex communications system, downstream data 
,5 packets may be transferred from a f.s. computer (or node, to a second 
computer (or node), and upstream data p^kets may be transferred from dre 
second computer (or node) to the first computer (or node). In such a scenarto, 
.he classifier 260 may consider both the downstream and upstream data 

,„ ,h, same flow For example, assume that the first 
packets to correspond to the same now. 

,0 computer has an IP address "A" and the second computer has an IP address 
..B" If a first data packet is tmnsmitted downstream from the firs, computer 
.„ the second computer, the first header HDR of the first data packet has a 
source IP address equals "A" and has a destination IP address that equals 
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"B". As described above, the first header HDR is extracted from the data 
packet via the data path unit 240 and suppUed to the classifier 260, and upon 
evaluating the first header HDR, the classifier 260 classifies the packet as 
belonging to the first flow. On the other hand, if a second data packet is 
5 transmitted upstream from the second computer to the first computer, for 
example for the purpose of acknowledging receipt of such first packet by the 
second computer, the second header HDR of the second data packet would 
have a source IP address that equals "B" and would have a destination IP 
address that equals "A". In such case, the data path unit 230 extracts the 

10 second header HDR from the data packet and outputs it to the classifier 260 as 
described above. Then, the classifier 260 evaluates the second header HDR 
and also classifies the second packet as belonging to the first flow. In other 
words, the classifier 260 classifies both the first and second data packets as 
belonging to the first flow because they are part of the same full duplex 

15 communication between the first and second computers. 

In the above example, the classifier 260 determines the flow to which a 
data packet belongs based on the source and destination IP addresses 
contained in the header HDR of the data packet. In addition to determining 
the flow of a data packet based on the IP addresses, the classifier 260 may also 

20 determine the flow based on the source and destination port values contained 
in the header HDR of the data packet. For example, in a full duplex 
communication system, if a first dovmstream data packet has a source IP 
address "A", a source port value "Al", a destination IP address "B", and a 
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destination port value "Bl", the classifier 260 may classify the first 
downstream data packet as belonging to a first flow. Similarly, a first 
upstream data packet having a source IP address "B", a source port value 
"Bl", a destination IP address "A", and a destination port value "Al" may still 
5 be classified as belonging to the first flow. However, a second downstream 
data packet having a source IP address "A", a source port value "A2", a 
destination IP address "B", and a destination port value "B2" may be 
classified as belonging to a second flow. Furthermore, the classifier 260 can 
p additionally or alternatively identify a specific flow of the data packet based 

3 10 on the protocol value contained in the header HDR of the data packet. 

- Each of the packet processors PPl to PPN is designated to process data 

packets belonging to a particular flow. For example, a first packet processor 
S (e.g. the processor PPl) may be designated to process data packets belonging 

g to a first flow, and a second packet processor (e.g. the processor PP2) may be 

3 15 designed to process data packets belonging to a second flow. Since the packet 

processor PPl is the only processor that processes data packets belonging to 
the first flow, it performs the same or similar types of processing functions for 
such data packets. As a result, the processor PPl is able to predict the types of 
data packets that it will receive and predict the types of processing operations 
20 it will have to perform. Thus, the processor PPl is capable of utilizing parallel 
processing, data cache hits, and other types of processing techniques to 
increase its processing speed and efficiently and quickly process the data 
packets corresponding to the first flow. Similarly, since the packet processor 
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PP2 is the only processor that processes data packets belonging to the second 
flow, it is able to process all of the data packets belong to the second flow in 
an extremely quick and efficient manner. 

Also, as described above, upstream and downstream data packets that 
5 belong to the same flow are processed by a single packet processor PPl, PP2, . 
. or PPN. As a result, two different packet processors are not required to 
separately process the upstream data packets and the downstream data packets. 
Thus, fewer processors PPl to PPN are required to process the data packets 

□ transmitted on the network, and the overall processing operation of the 

'.f\ 

;3 10 network is enhanced. In addition, as shown in Fig. 2, the packet processors 

PPl to PPN are connected to the data path units 230 and 240 via a data bus. 

''^ As a result of such architecture, packet processors can easily be added to or 

removed from the system. Thus, as the number of process flows handled by 

p the system 200 increases (or decreases), processors can be simply added to (or 

15 removed from) the system 200 to accommodate the change in the number of 
process flows. 

The classifier 260 designates which packet processors PPl to PPN 
process data packets corresponding to the various flows and instructs the data 
path units 230 and 240 to output the data packets to the appropriate processor 
20 PPl, PP2, . . ., or PPN. In one embodiment, the classifier 260 designates 
which processors PPl to PPN will process the data packets, in the following 
manner. First, as described above, the classifier 260 receives a particular 
header HDR of a particular data packet and determines that the particular data 
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packet corresponds to a particular flow. Then, the classifier 260 determines if 
any of the packet processors PPl to PPN have previously been designated to 
process data packets belonging to the particular flow. If the classifier 260 
determines that one of the packet processors PPl to PPN has been previously 
5 designated to process the particular flow, the classifier 260 determines that the 
particular data packet should be output to the particular data processor. 

On the other hand, if none of the processors PPl to PPN have been 
previously designated to process the particular flow, the particular flow is 
considered to be a new flow. In such case, the classifier 260 selects one of the 

10 processors PPl to PPN to process data packets belonging to the new flow 
based on the processing load of the processors PPl to PPN and determines that 
the particular data packet, and any subsequent data packet belonging to the 
same process flow, should be output to the selected data processor. 

In one embodiment, the classifier 260 receives, in real time, processing 

15 load information from the packet processors PPl to PPN via the Load 
Information Bus illustrated in Fig. 2, and such information indicates the 
processing loads of the processors PPl to PPN. Upon receiving the processing 
load information, the classifier 260 determines the processing load of the 
processors PPl to PPN based on such information. Then, the classifier 260 

20 assigns the new flow to the packet processors PPl to PPN based on their 
processing load. 

For example. Fig. 3 illustrates processing load information that is 
output from each of the packet processors PPl to PPN based on its processing 
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load. In particular, if a particular packet processor (e.g. PPl) has an extremely 
heavy processing load and cannot effectively process any additional data, it is 
deemed to be "unavailable". Thus, it outputs a "00" to the classifier 260 as the 
processing load information. Upon receiving such information, the classifier 
260 determines that it cannot designate the processor PPl to handle any new 
process flows. If the processor PPl has a very heavy processing load, it is 
deemed to be "heavily loaded". In such case, the processor PPl outputs a 
"01" to the classifier 260 as the processing load information. Based on such 
information, the classifier 260 determines that it can only assign a few more 
new process flows to the processor PPl. If the processor PPl has an average 
processing load, it is deemed to be "moderately loaded". In^such instance, the 
processor PPl outputs a "10" to the classifier 260 as the processing load 
information. Upon receiving such information, the classifier 260 determines 
that it can occasionally assign new process flows to the processor PPl. 
Finally, if the processor PPl has a relatively light processing load, it is deemed 
to be "lightly loaded". As a result, the processor PPl outputs a "11" to the 
classifier 260 as the processing load information. When the classifier 260 
receives such information, it determines that the processor PPl is available to 
handle as many new process flows as necessary. 

As shown in Fig. 3, the processing load information is divided into 
four categories. However, the number of categories is clearly not limited to 
four, and the processing load information may be divided into more or less 
categories based on the specific application of the invention. 
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After receiving the processing load information from the processors 
PPl to PPN, the classifier 260 is able to assign the new process flow to the 
processors PPl to PPN such that the processing loads of the processors PPl to 
PPN are evenly balanced. As a result, the overall efficiency of the system 200 
is enhanced because potential bottlenecks of data packets at overloaded 
processors are eliminated and because the resources of the system 200 are 
fiilly utilized in the most efficient manner possible. Moreover, the classifier 
260 is always updated, in real-time, of the current load information of each of 
the packet processors. 

In one implementation, the classifier 260 evaluates the processing load 
information and determines which processor PPl,..., or PPN has the smallest 
processing load. Then, the classifier 260 assigns the new process flow to the 
packet processor PPl,..., or PPN having the smallest load. If more than one 
of the processors PPl to PPN are categorized as having the smallest 
processing load, the classifier 260 sequentially assigns new process flows all 
of the processors falling within such category. For instance, if the processors 
PP2, PP5, and PP6 are all categorized as having the smallest processing load 
(e.g. the processors PP2, PP5, and PP6 output processing load information that 
equals "1 1")? the classifier 260 may initially assign a new process flow to the 
processor PP2. Then, the classifier 260 may assign the next new process flow 
to the processor PP5. Afterwards, when another new process flow needs to be 
assigned, the classifier 260 may assign the process flow to the processor PP6. 
The classifier 260 may continue to repeatedly and sequentially assign new 
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process flows to the processors PP2, PP5, and PP6 until the load of one of the 
processors (e.g. the processor PP5) increases such that the processor PP5 is no 
longer classified as having the smallest processing load (e.g. the processing 
load information output from the processor PP5 equals "10"). In such case, 
the classifier 260 continues to repeatedly and sequentially assign new process 
flows to only the processors PP2 and PP6. 

In the example described above, the classifier 260 identifies all of the 
packet processors that are categorized as having the smallest processing load 
(e.g. processors PP2, PP5, and PP6) and sequentially assigns new flows to the 
processors PP2, PP5, and PP6 in a particular order. However, the present 
invention is clearly not limited to assigning flows to the processors PP2, PP5, 
and PP6 in any particular order, and the classifier 260 may randomly assign 
new flows to the processors PP2, PP5, and PP6. Furthermore, the classifier 
260 may not assign new flows to the processors PPl to PPN based only on the 
processing load information and may evaluate other information to determine 
which one of the processors PPl to PPN to designate to handle the flow. For 
instance, a subset of the processors PPl to PPN may be specifically adapted to 
process a particular type of -flow. In such case, if a new flow is received and is 
the particular type of flow, the classifier 260 may evaluate the processing load 
information received from the subset of the processors PPl to PPN and 
determine which processor within the subset of the processors PPl to PPN 
should process the new flow. 
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In another implementation, the classifier 260 does not assign new 
flows to only the packet processors PPl to PPN that are categorized as having 
the smallest load. Specifically, the classifier 260 analyses the processing load 
information of the packet processors PPl to PPN and assigns new flows to the 
5 packet processors PPl to PPN with a frequency that is based on the load 
information of the processors PPl to PPN. An illustrative example of how the 
classifier 260 assigns new flows in such manner is described below in 
conjunction with Fig. 4. 

As shown in Fig. 4, eight packet processors PPl to PP8 are contained 

10 in the system 200. During the first round of assigning new process flows to 
the packet processors PPl to PP8, the classifier 260 only assigns new flows to 
the processors that have the smallest processing load. In the present example, 
the classifier 260 assigns new flows to the processors having processing load 
information that equals "1 1". Thus, the classifier 260 sequentially assigns the 

15 first and second process flows to the processors PPl and PP6, respectively. 

During the second round of assigning new process flows to the packet 
processors PPl to PP8, the classifier 260 only assigns new flows to the 
processors that have the smallest processing load and that have the second 
smallest processing load. In the present example, the classifier 260 assigns 

20 new flows to the processors having processing load information that equals 
"11" and "10". Thus, the classifier 260 sequentially assigns the third to sixth 
new process flows to the processors PPl, PP2, PP6, and PP7, respectively. 
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During the third round of assigning new process flows to the packet 
processors PPl to PP8, the classifier 260 only assigns new flows to the 
processors that have the smallest processing load £ind that have the third 
smallest processing load. In the present example, the classifier 260 assigns 
5 new flows to the processors having processing load information that equals 
"11" and "01". Thus, the classifier 260 sequentially assigns the seventh to 
eleventh new process flows to the processors PPl, PP3, PP4, PP6, and PP8, 
respectively. 

During the fourth round of assigning new process flows to the packet 
10 processors PPl to PP8, the classifier 260 only assigns new flows to the 
processors that have the smallest processing load and that have the second 
smallest processing load. In the present example, the classifier 260 assigns 
new flows to the processors having processing load information that equals 
"11" and "10". Thus, the classifier 260 sequentially assigns the twelfth to 
15 fifteenth new process flows to the processors PPl, PP2, PP6, and PP7, 
respectively. 

As shown in the above example, the classifier 260 assigns new process 
flows to the packet processors having the smallest processing load (i.e. the 
processors PPl and PP6) every round. Also, the classifier 260 assigns new 
20 process flows to the packet processors having the second smallest processing 
load (i.e. the processors PP2 and PP7) every second round. In addition, the 
classifier 260 assigns new process flows to the packet processors having the 
third smallest processing load (i.e. the processors PP3, PP4, and PP8) every 
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third round. Finally, the classifier does not assign new process flows to the 
"unavailable" processors (i.e. processor PP5) during any round. As a result, 
the frequency with which the classifier 260 assigns new process flows to the 
processors PPl to PP8 is based on the processing loads of the processors PPl 
to PP8. 

Although in the above embodiment, the classifier 260 assigns new 
process flows to the packet processors PPl to PP8 in the manner described 
above and illustrated in Fig. 4, the embodiment is not limited to such method 
of assignment. For example, the classifier 260 may assign new flows to the 
processors PPl to PP8 in any manner such that new process flows are assigned 
to the processors PPl to PP8 at a frequency that corresponds to their 
processing loads. 

In another embodiment, the processing load information output from 
the packet processors PPl to PPN does not contain the data shown in Fig. 3. 
Instead, each of the processors PPl to PPN may output processing load 
information that indicates the number of processing flows it is currently 
handling. Then, the classifier 260 may determine the relative loads of the 
processors PPl to PPN based on the load information and the total number of 
flows that have been assigned to all of the processors PPl to PPN. 

For example, the classifier 260 may keep track of the total number of 
process flows that have been assigned to all of the processors PPl to PPN and 
may calculate percentage of the process flows being handled by each of the 
processors PPl to PPN based on the processing load information received 
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from the processors PPl to PPN. For instance, assume that four processors 
PPl to PP4 are contained in the system 200 and that a total of 100 process 
flows have been assigned to the processors PPl to PP4. Also, assume that the 
processing load information indicates that the first processor PPl is handling 
5 50 process flows, the second processor PP2 is handling 25 process flows, the 
third processor PP3 is handling 1 5 process flows, and the fourth processor PP4 
is handling 1 0 process flows. In such a scenario, the classifier 260 can easily 
determine that the relative processing loads of the processors PPl to PP4 are 
J 50%, 25%, 15%, and 10%, respectively. 

m 

i;3 10 In the above example, the classifier 260 may determine which 

rU processor PPl, PP2, PP3, or PP4 has the smallest processing load by directly 

evaluating the relative processing load percentages. On the other hand, the 
classifier 260 may categorize the processing loads of the processors PPl to 
: 5 PP4 into categories based on the percentages. For instance, in the above 

□ 15 example, the classifier 260 may categorize the processing loads of the 

processors PPl to PP4 based on the table shown in Fig. 5. According to the 
example above, the processor PPl (with a relative processing load percentage 
of 50%) would be considered to be "heavily loaded", the processor PP2 (with 
a relative processing load percentage of 25%) would be considered to be 
20 "moderately loaded", and the processors PP3 and PP4 (with relative 
processing load percentages of 15% and 10%) would be considered to be 
"lightly loaded". Then, the classifier 260 may assign new flows to the 
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processors PPl to PP4 in a manner that is similar to any one of the manners 
described above in conjunction with Figs. 3 and 4. 

Also, as described above, the number of categories shovm in Fig. 5 for 
classifying the loads of the processors equals four. How^ever, the number of 
categories clearly can be greater than or less than four depending on the 
particular application of the invention. 

In another embodiment, the packet processors PPl to PPN output 
processing load information that corresponds to the time that the processors 
PPl to PPN are in an idle mode (i.e. not performing any substantive tasks). 
An example of when the processors PPl to PPN are in an idle mode is when 
they are performing certain memory clean-up activities. When the classifier 
260 receives the processing load information from the processors PPl to PPN, 
it is able to evaluate the information to determine the processing load of the 
processors PPl to PPN. Specifically, if the processing load information 
indicates that a particular packet processor is in an idle mode very frequently 
and/or for long periods of time, the classifier 260 considers the processing 
load on the processor to be relatively small. In other words, the processing 
load of the particular packet processor is inversely related to the amount of 
time that the particular packet processor is in the idle mode. 

In the present embodiment, the classifier 260 may determine which 
processor PPl, PP2, PP3, or PP4 has the smallest processing load by directly 
evaluating the amount of time that the processors PPl to PP4 are in the idle 
mode. On the other hand, the classifier 260 may categorize the processing 
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loads of the processors PPl to PP4 into the categories shown in Fig. 6 based 
on the amount of time that the processors are in the idle mode. For instance, 
in the above example, the classifier 260 (1) may consider a processor to be 
"unavailable" if it has been in the idle mode for less than an amount of time A, 

(2) may consider the processor to be "heavily loaded" if it has been in the idle 
mode for greater than the amoimt of time A and less than an amount of time B, 

(3) may consider the processor to be "moderately loaded" if it has been in the 
idle mode for greater than the amount of time B and less than an amount of 
time C, and (4) may consider the processor to be "lightly loaded" if it has been 
in the idle mode for greater than the amount of time C. The amounts of time 
A, B, and C shown in Fig. 6 may correspond to actual times, a cumulation of 
times over a certain time period, or percentages of times over a certain time 
period. Also, the amounts of time A, B, and C may correspond to other 
measurements of time. After the processing loads of the processors PPl to 
PP4 are classified into the categories shown in Fig. 6, the classifier 260 may 
assign a new flow to the processor in a manner that is similar to any one of the 
manners described above in conjunction with Figs. 3 and 4. 

As described above, the number of categories shown in Fig. 6 for 
classifying the loads of the processors equals four. However, the number of 
categories clearly can be greater than or less than four depending on the 
particular application of the invention. 

In an additional embodiment, the classifier 260 determines the 
processing load of the packet processors PPl to PPN by determining the 
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amount of data that is routed to the processors PPl to PPN. For example, the 
classifier 260 may determine the processing load of a particular packet 
processor (e.g. processor PPl) by determining the number of bits per second 
that is supplied to the processor PPL The number of bits per second supplied 
5 to the processor PPl may be based on the number of bits supplied to the 
processor PPl over a predetermined period of time or may be a running 
average of the number of bits being continuously supplied to the processor 
PPl. If the rate at w^hich data is supplied to the processor PPl is relatively 
large, the classifier 260 determines that the processing load of the processor 

10 PPl is relatively high. On the other hand, if the rate at which data is supplied 
to the processor PPl is relatively small, the processing load of the processor 
PPl is relatively low. In other words, the rate at which data is supplied to the 
processor PPl is directly related to the processing load of the processor PPl . 

In the embodiment described above, the classifier 260 may determine 

1 5 the processing loads of the processors PP 1 to PPN by directly evaluating the 
rate at which data is supplied to the processors PPl to PPN. Alternatively, the 
classifier 260 may categorize the processing loads of the processors PPl to 
PPN into the categories shown in Fig. 7 based on the rate at which data is 
supplied. For example, in the above example, the classifier 260 (1) may 

20 consider a processor to be "unavailable" if the rate at which data is supplied to 
the processor exceeds a data rate A, (2) may consider the processor to be 
"heavily loaded" if the rate at which data is supplied to the processor is less 
than the data rate A and greater than a data rate B, (3) may consider the 
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processor to be "moderately loaded" if the rate at which data is supplied to the 
processor is less than the data rate B and greater than a data rate C, and (4) 
may consider the processor to be "lightly loaded" if the rate at which data is 
supplied to the processor is less than the data rate C. Then, the classifier 260 
5 may assign a new flow to the processor in a manner that is similar to any one 
of the manners described above in conjunction with Figs. 3 and 4. 

Also, as mentioned above, the number of categories shovm in Fig. 7 
for classifying the loads of the processors equals four. However, the number 
of categories clearly can be greater than or less than four depending on the 

10 particular specific application of the invention. 

In the embodiments described above in conjunction with Figs. 3, 6, and 
7, a unique situation arises when the processing load of all of the packet 
processors PPl to PPN in the system 200 is extremely heavy and all of the 
processors PPl to PPN are categorized as "unavailable". In such a situation, if 

15 the classifier 260 continued to operate according to the examples described 
above, the classifier 260 would not be able to assign any new flows to any of 
the processors PPl to PPN. On the other hand, in a preferred embodiment, the 
classifier avoids such a problem in the following manner. 

First, when the classifier 260 determines that all of the packet 

20 processors PPl to PPN in the system are overloaded and "unavailable", the 
classifier 260 resets the status of all of the processors PPl to PPN (or instructs 
the packet processors PPl to PPN to reset their own status) to the "lightly 
loaded" status. Then, the classifier 260 assigns new process flows to the 
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packet processors PPl to PPN in accordance with any of the embodiments 
described above. 

In an alternative implementation of the embodiment, the classifier 260 
does not reset the status of all of the packet processors PPl to PPN (or does 
5 not instruct all of the packet processors PP 1 to PPN to reset their own status) 
to the "lightly loaded" status. Instead, the classifier 260 readjusts the 
threshold levels of load balancing system by scaling the threshold levels of the 
load balancing system. For example, assume that the load balancing system 
contains four packet processors PPl to PP4, and the processing loads of the 

10 processors PPl to PP4 are determined in accordance with the processing load 
percentages contained in the load balancing table shown in Fig. 5. If the loads 
of the processors PPl to PP4 equal 75%, 85%, 90% and 95%, respectively, all 
of the processors PPl to PP4 are deemed to be "unavailable". In such a 
scenario, the classifier 260 resets the status of at least one of the processors 

15 PPl to PP4 (or instructs at least one of the processors PPl to PP4 to reset its 
own status) so that at least one of the processors PPl to PP4 does not have the 
"unavailable" status. For instance, the classifier 260 may reclassify the 
processing loads of the processors PPl to PP4 such that a processing load 
percentage of 0% to 79% corresponds to the "lightly loaded" status, a 

20 processing load percentage of 80% to 89% corresponds to the "moderately 
loaded" status, a processing load percentage of 90% to 95% corresponds to the 
"heavily loaded" status, and a processing load percentage of 96% to 100% 
corresponds to the "imavailable" status. Based on the reclassified processing 
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loads, the processor PPl is deemed to be "lightly loaded", the processor PP2 is 
deemed to be "moderately loaded", and the processors PP3 and PP4 are 
deemed to be "heavily loaded". Afterwards, the classifier 260 assigns new 
flows to the processor PPl to PP4 based on the readjusted processing loads. 
Also, if the processing load percentages of all of the processors PPl to PP4 
fall below 80% so that all of the processors PPl to PP4 are deemed to be 
"lightly loaded", the classifier 260 may again reclassify the processing loads 
of the processors PPl to PP4 in accordance with the values contained in the 
table shown in Fig. 5. 

Clearly, the present embodiment is not limited to the specific example 
described above. For example, those skilled in the art would easily be able to 
apply the above embodiment to other types of load balancing systems. In 
addition, the classifier 260 could reclassify the processing loads of the 
processors PPl to PP4 in accordance with different ranges of processing load 
percentages. 

The previous description of the preferred embodiments is provided to 
enable a person skilled in the art to make or use the present invention. 
Moreover, various modifications to these embodiments will be readily 
apparent to those skilled in the art, and the generic principles defined herein 
may be applied to other embodiments without the use of inventive faculty. 
For example, in most of the embodiments described above, new process flows 
are allocated to the various packet processors PPl to PPN such that the 
processing loads of the processors PPl to PPN are approximately equal. 
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However, such embodiments can be modified such that the new process flows 
are assigned to the processors PPl to PPN in a manner that the processing 
loads of the processors PPl to PPN have a certain ratio with respect to each 
other. Alternatively, the new process flows can be assigned such that the 
processing loads of the processors PPl to PPN do not exceed particular 
thresholds respectively assigned to the processors PPl to PPN, and such 
thresholds may or may not have different values. Therefore, the present 
invention is not intended to be limited to the embodiments described herein 
but is to be accorded the widest scope as defined by the claims and equivalents 
thereof 

In addition, various operations recited in the claims are designated 
alphabetically. However, such designations are not intended to indicate the 
order in which the operations are to be performed." For example, the 
alphabetically designated operations can be performed in different orders than 
the alphabetized order and can even be performed simultaneously in some 
instances. 
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